Skip to content

Adding atunnel - networking component within ateom - #559

Merged
Bowei Du (bowei) merged 7 commits into
agent-substrate:mainfrom
LiorLieberman:pr1-atunnel-ingress
Jul 31, 2026
Merged

Adding atunnel - networking component within ateom#559
Bowei Du (bowei) merged 7 commits into
agent-substrate:mainfrom
LiorLieberman:pr1-atunnel-ingress

Conversation

@LiorLieberman

@LiorLieberman Lior Lieberman (LiorLieberman) commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

[Will rebase after its lgtm]

Introducing atunnel, a component that sits within ateom and handles all inbound and outbound traffic to the actor. With this change in place there is no direct traffic outside and inside the actor with the exception of DNS and non-TCP which traffic which is still TBD.

this PR;

  • Introduces atunnel, on ingress, it terminates tls and forward to the actor on port 80 (We'll have separate discussions on supporting multiple ports). On egress, it initiates CONNECT over mtls.
  • Currently atunnel is using the pod/worker identity for mtls, which is wrong. We should likely use the actor identity once Taahir Ahmed (@ahmedtd) work is done on the SessionIdentity (likely to be renamed to something else) to be able to mint certs and JWTs.
  • RunRequest/RestoreRequest has actor_version and egress_gateway_address here because ateom and atunnel consume them in the same functions that install the nftables rules. Nothing populates either field in this PR, so they arrive zero-valued, so the egress client is never built and no REDIRECT rule is installed. A fast follow-up egress gateway PR adds the producer side that configures these fields.

Huge part of it uses and is built-upon John Howard (@howardjohn) 's work on #460


More specific notes:

  • New internal/atunnel package — an mTLS ingress server plus a transparently-intercepted egress TCP proxy. Only the ingress half is live in this PR.

  • Every worker pod hosts an atunnel ingress server on :443. The server authorizes the caller and checks the Actor is the one currently assigned to that worker.

  • atenet-router dials the tunnel instead of the Actor. The dynamic_forward_proxy cluster is replaced by an ORIGINAL_DST cluster driven by an ext_proc-set x-ate-original-dst header, with upstream mTLS and SPIFFE URI SAN validation. :authority is left untouched, so atunnel can authorize on the original Host.

  • The pod-IP:80 → actor-veth:80 DNAT is removed. Worker port 80 is no longer an ingress path; the mTLS listener is the only way in.

  • The egress proxy, the nftables REDIRECT rule and the two new ateompb fields are all gated on an egress gateway address that nothing populates yet, so Actor egress still uses the existing masquerade, unchanged. A pluggable-egress PR is a fast follow.

  • Tests pass

  • Appropriate changes to documentation are included in the PR

@haiyanmeng

Copy link
Copy Markdown
Collaborator

FYI, #557 maps HTTPS requests to actor port 443 with TLS re-origination- #557

Comment thread docs/architecture.md
extracts the Actor name and Atespace from the `Host` header and calls the
Control Plane to resume the Actor and resolve its current worker assignment.

* **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change this to Worker Ingress Tunnel

Comment thread docs/architecture.md
Comment thread docs/architecture.md Outdated
Comment thread docs/architecture.md
extracts the Actor name and Atespace from the `Host` header and calls the
Control Plane to resume the Actor and resolve its current worker assignment.

* **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worthy adding a one-line like egress redirection through atunnel is implemented but not yet enabled, which would stop the readers from concluding either that it's live or that it doesn't exist.

Comment thread internal/atunnel/client.go
Comment thread cmd/atenet/internal/router/cmd.go Outdated
Comment thread cmd/atenet/internal/router/cmd.go Outdated
Comment thread docs/threat-model.md

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review that before we merge - this is ai generated fully

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

text changes look ok to me

@LiorLieberman
Lior Lieberman (LiorLieberman) force-pushed the pr1-atunnel-ingress branch 3 times, most recently from a0c14f2 to 6030db8 Compare July 29, 2026 01:28
Comment thread cmd/atenet/internal/router/config.go
signerName: podidentity.podcert.ate.dev/identity
labelSelector:
matchLabels:
podcert.ate.dev/canarying: live

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: not sure if we need this or not. Discuss with other community members

}

// The actor is reached through the in-worker atunnel ingress server, which
// listens on :443 (mTLS) and forwards to the actor's :80. The worker no

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking we should use not 443, but own port just in case we need the standard HTTPS port later.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline but commeting for posterity. This port is standard https.

When we introduce CONNECT on Ingress (client -> atenet/pluggable ingress) - we may also have connect to atunnel, and in that case, we can consider having that a 8443 or any other special port.

Comment thread cmd/ateom-gvisor/main.go
Comment thread cmd/ateom-gvisor/main.go Outdated
Comment thread cmd/ateom-gvisor/main.go
return nil
}

func runAtunnel(ctx context.Context, upstream *url.URL) (*atunnel.Server, *atunnel.Egress, uint16, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this going to be same or dupe for ateom-uvm

func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.CheckpointWorkloadRequest) (*ateompb.CheckpointWorkloadResponse, error) {
s.lock.Lock()
defer s.lock.Unlock()
if err := s.deactivateActorNetworking(ctx); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this just a bug that it's missing on the CheckpointWorkload path? Specifically, we should just make this change outside of this change.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deactivateActorNetworking is a new function this PR added. Ideally we are trying to stop admitting traffic and drain active streams before the Actor network is torn down. Since (1) we introduced atunnel and it takes care of all networking connections for the actor and (2) CheckpointWorkload suspends the actor. I think the best home for this addition is within this PR.

Comment thread internal/atunnel/client.go
Comment thread internal/atunnel/client.go Outdated
Comment thread internal/atunnel/client.go Outdated
}

// Client opens actor egress streams through an mTLS-authenticated gateway.
type Client struct {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would be good to make this more consistent, calling everything egress related egress vs a very generic sounding Client.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally did it and then reverted, there is nothing egress-only in client. Client implements EgressDialer though.. Suggest we leave it this way

Comment thread internal/atunnel/client.go
Comment thread internal/atunnel/egress.go

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One Plausible AI review finding.

Comment thread internal/atunnel/server.go Outdated
return loadCredentialBundle(s.credentialBundlePath)
},
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: clientCAs,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 should-fix 🟡 – The client CA pool is frozen at process start while the leaf beside it reloads per connection. GetCertificate above re-reads the credential bundle on every handshake, with a comment explaining that this is what lets kubelet's rotation take effect — but clientCAs comes from a single os.ReadFile in NewServer.

Both files live in the same projected volume: workerpool_apply.go gives it a podCertificate source for credential-bundle.pem and a clusterTrustBundle source for trust-bundle.pem. A ClusterTrustBundle projection exists precisely so kubelet can push CA changes into a running pod, and ateom outlives many activations, so after a signer CA rotation this server keeps verifying against the old pool and rejects the router until the worker pod restarts.

GetConfigForClient is the usual fix: return a *tls.Config per connection with a freshly built ClientCAs, mirroring what GetCertificate already does for the leaf. client.go reads its RootCAs once too, though NewClient runs per activation, so the window there is much shorter.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lior can you take a TODO to follow up on this

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a todo.. do we have rotation today?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we should have it...otherwise substrate will stop working after 1 - 2 days due to the certs having very quick expiry.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sg, added a todo and will open an issue for it as well. Rebased now, hopefully all should be good to go

@LiorLieberman
Lior Lieberman (LiorLieberman) force-pushed the pr1-atunnel-ingress branch 2 times, most recently from aa4874d to 65c5e58 Compare July 31, 2026 17:31
atunnel is the in-worker component that carries Actor traffic in both
directions over authenticated channels.

  * Server: an mTLS HTTPS listener that terminates connections from the
    ingress gateway, authorizes the peer by its SPIFFE identity, checks
    that the request targets the Actor currently activated on this
    worker, and reverse-proxies to the Actor over the private veth.

  * Egress: an activation-aware TCP proxy for transparently intercepted
    Actor connections. It resolves the original destination via
    SO_ORIGINAL_DST and forwards through an EgressDialer, asserting the
    Actor identity to the far end. Dormant until an egress gateway is
    configured; the client and dialer land here so the package is
    reviewed as one unit.

Activate/Deactivate bracket an Actor activation so traffic is only
carried while an Actor is assigned to the worker, and Deactivate drains
in-flight streams before the Actor network is torn down.
Every worker pod now hosts an atunnel ingress server on :443 and an
atunnel egress listener, both long-lived, with per-activation
Activate/Deactivate bracketing every Run, Restore and Checkpoint.

The nftables rules ateom installs change accordingly:

  * The pod-IP:80 -> actor-veth:80 DNAT is gone. Worker port 80 is no
    longer an Actor ingress path; the only way in is the mTLS listener
    on :443, which authorizes the caller and checks the Actor is the one
    currently assigned to this worker.

  * A prerouting REDIRECT sends Actor TCP egress to the local atunnel
    egress listener, preserving SO_ORIGINAL_DST. It is only installed
    when the activation carries an egress gateway address, which nothing
    populates yet, so the masquerade path is unchanged and Actor egress
    behaves exactly as before.

The ateom Run/Restore protos gain the two fields that arm that path:
egress_gateway_address, which decides whether the redirect is installed
at all, and actor_version, the Actor resource version ate-api observed
when assigning the worker, which atunnel asserts to the egress gateway
as a lower bound on trustworthy Actor metadata. Both are consumed here
and left unset. atelet and ate-api start populating them in the egress
gateway change, which is the point at which they mean anything, so this
change adds no new requirement to the atelet wire contract.

atecontroller gives worker pods the podidentity credential + trust
bundles (the atunnel server identity), the servicedns trust bundle, and
container port 443. podidentitysigner now issues certs with
ExtKeyUsageServerAuth as well as ClientAuth, without which the worker
cannot present its podidentity cert as a TLS server cert and the
gateway handshake fails.
The router used to rewrite :authority to the actor's worker pod IP and let
the dynamic_forward_proxy cluster resolve it, reaching the actor over
plaintext pod-IP:80. The worker no longer DNATs pod-IP:80 to the actor, so
that path is gone.

Instead the ext_proc resolves the actor to its worker's atunnel ingress
address (IP:443) and puts it in x-ate-original-dst. An ORIGINAL_DST cluster
dials exactly that address, which leaves the request Host as the actor DNS
name -- atunnel needs it to authorize the active actor. The header is set
with OVERWRITE_IF_EXISTS_OR_ADD so a client-supplied value can never
influence the address Envoy dials.

The upstream hop is mTLS: the cluster presents the router's podidentity
credential bundle as its client cert and validates the atunnel server
against the podidentity trust bundle. Validation matches the SPIFFE URI SAN
prefix rather than the dialed pod IP, because the atunnel cert carries only
a spiffe:// URI SAN and Envoy's default SAN check against an ephemeral pod
IP would never match. atunnel in turn only accepts
spiffe://cluster.local/ns/ate-system/sa/atenet-router.

The dynamic_forward_proxy cluster, HTTP filter and DNS cache config are
removed along with the :authority rewrite.
Bring the architecture and threat-model docs in line with the atunnel
ingress path: the router no longer rewrites :authority to a worker pod IP
and forwards over plaintext port 80, it opens an mTLS tunnel to atunnel on
worker port 443, which forwards to the Actor over its private veth.

Drop cmd/atenet/atenet-diagram.png. It predates the ext_proc/ORIGINAL_DST
design and is now wrong in the part that matters most. The README section it
illustrated gains a short accurate note about the upstream hop instead of a
dangling image reference.

Add an e2e suite covering the change end to end: TestActorDirectAccess
asserts that the worker pod's port 80 is no longer a reachable Actor ingress
path (the DNAT rule is gone) and that the same Actor still answers /readyz
through atenet-router over the atunnel mTLS hop. It uses the counter demo as
its fixture, so it only needs --deploy-demo-counter.

internal/e2e/testmain.go picks up ParseSkippedFlags so that `go test` flags
(-run, -v, ...) survive pflag parsing and reach the suite.
@bowei
Bowei Du (bowei) merged commit 860250b into agent-substrate:main Jul 31, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants